Analyzing Happiness Around The World¶
Pauline Comising and Amanda Ma
Introduction¶
With the current state of the world, we wanted to focus on a more cheerful topic: happiness. More specifically, what contributes to our overall happiness, and how does that differ across different parts of the world? We found some interesting datasets to help us answers questions like this in this tutorial.
We will be guiding you through the data science pipeline, walking through three main sections:
- Data curation, parsing, and management
- Exploratory Data Analysis
- Prediction & Machine Learning
Through these steps, we will process, analyze, and draw conclusions from our data, hopefully bringing insight about what people across the globe view as important to their happiness. To learn more about the data science pipeline here are additional resources:
Our Questions¶
Through our analysis, we strive to answer the following questions:
- What regions are the happiest?
- What do countries find most important to be happy?
- How do real factors (like GDP and life expectancy) affect happiness and what people consider most important?
- Can weighing certain aspects as less important, lead to more happiness despite harsh realities such as low life expectancy?
And we look to data science for these problems as a way to:
- Take three different datasets and tell a story that goes beyond each of them individually (specifically question 4)
- Reveal a variety of different insights answering all the questions by being able to select and filter data from our master dataset
- Verifying hypotheses and making predictions through machine learning, specifically linear regression
- And in general: make sense of data that is otherwise scattered and hard to understand
Data Curation, Parsing & Management¶
First, lets import the libraries we will be using in this section:
import pandas as pd
Next, we will load our datasets. Our primary dataset (2019.csv) displays countries, their happiness score out of 10, and various factors that contribute to their overall happiness score. We also have two additional datasets (countries of the world.csv and life-expectancy.csv) that contain more information including actual GDP per capita and actual life expectancy for each country. We will be merging the data together.
data = pd.read_csv('2019.csv')
data
gdp = pd.read_csv('countries of the world.csv')
gdp
life_expec = pd.read_csv('life-expectancy.csv')
life_expec
For our two supplementary datasets, we will only select the data that we will be using in our further analysis and visualizations. More specifically, we are keeping the country name, region, and GDP per capita, and life expectancy.
gdp = gdp[['Country', 'Region', 'GDP ($ per capita)']]
gdp
Since the life expectancy dataset had data for each country, and for multiple years, we are also narrowing down our data to only focus on countries in the year 2019, as we want the most up to date analysis.
life_expec = (life_expec[life_expec['Year'] == 2019][['Entity', 'Life expectancy (years)']]
.rename(columns={'Entity':'Country'})
.reset_index())
life_expec = life_expec[['Country', 'Life expectancy (years)']]
life_expec
From closer inspection, we realized that there was an extra space after each country name in the second dataset. We are getting rid of that so we can match the Country columns, and prepare to merge the datasets.
This brings up an important point. As a data scientist, one should know their data quite well, including how it is formatted. Knowing these details will make it easier to clean and manage data, and even for visualizations/further analysis.
countries = gdp['Country'].tolist()
new_countries = []
for c in countries:
new_countries.append(c.strip())
gdp['Country'] = pd.DataFrame({'Country': new_countries})
Now, we will join the datasets on country name. We are using the key word 'inner', meaning that when we merge the data, we will only match the countries and corresponding data that exist in all three datasets.
data = data.rename(columns={"Country or region": "Country"})
data = data.merge(gdp, on='Country', how='inner')
data = data.merge(life_expec, on='Country', how='inner')
data
Now that we've combined our datasets, we see that some column titles are similar. To avoid confusion, let's clean them up and rename them for better clarity.
data = data.rename(columns={"Score": "Happiness Score", "GDP per capita": "GDP Importance"
, "Social support": "Social Support Importance"
, "Healthy life expectancy": "Healthy Life Expectancy Importance"
, "Freedom to make life choices": "Freedom Importance"
, "Generosity": "Generosity Importance"
, "Perceptions of corruption": "Absence of Corruption Importance"
, "GDP ($ per capita)": "Actual GDP"
, "Life expectancy (years)": "Actual Life Expectancy"})
data
Exploratory Data Analysis¶
Now that we have loaded and tidied our data, we will move on to creating visualizations, and drawing conclusions from them. We will be using different visualization libraries, so we can show you different tools and ways you can display your data!
For our visualizations, we will be using Seaborn, Plotly, Matplotlib, and Numpy.
import seaborn as sns
import plotly.graph_objs as go
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import matplotlib.pyplot as plt
import numpy as np
To get a better idea of which parts of the world have greater overall happiness, we will create boxplots using Seaborn, comparing different regions' happiness scores.
ax = sns.boxplot(x = 'Region', y = 'Happiness Score', data = data)
ax.set_title("Happiness Across Regions Around the World")
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
plt.show()
From this plot, we can get a clear idea of which regions are generally the happiest. We see that Western Europe, Oceania, and North America are leading with the greatest happiness scores. The boxplots also show the variability amongst countries in the specified region. For example, the 'Near East' region has the greatest range, showing that there are countries that have a very high happiness scores, but also some that have quite a low score.
Another way to display which countries/regions are happier is through a map, using Plotly. We will create a choropleth map, with a range of colors indicating the happiness score of each country listed in the dataset.
d = dict(type = 'choropleth',
locations = data['Country'],
locationmode = 'country names',
z = data['Happiness Score'],
text = data['Country'],
colorbar = {'title':'Happiness'})
layout = dict(title = 'Happiness Across the World',
geo = dict(showframe = False))
cmap = go.Figure(data = [d], layout=layout)
iplot(cmap)
This map creates a great visual for analysis, as we can compare different regions' happiness, and specific countries as well. We can also hover over specific areas to see the country name and their happiness score, which may make this visualization even more useful and detailed than the first one. Looking at the map, we can see that North America, Europe, and Australia seem to be leading in the happiness ranking, which is similar to our initial conclusions from the boxplots. The variability in regions that we pointed out in the first plot can now be seen more clearly as well; for example, we can see how some Eastern and Asian countries have low happiness, while others are much happier.
Now that we've analyzed which countries/areas have the greatest happiness rankings, we can move into the 'why?'. What contributes to their happiness? For closer analysis, we will separate the countries into 3 bins: countries with high happiness, medium happiness, and low happiness. We will compare which factors are more important to the countries in these three categories, and whether or not they are similar or different based on their current happiness levels.
data['General Happiness Level'] = pd.cut(data["Happiness Score"], 3
, labels=["Low", "Medium", "High"])
low_happiness = data.loc[data['General Happiness Level'] == "Low"]
medium_happiness = data.loc[data['General Happiness Level'] == "Medium"]
high_happiness = data.loc[data['General Happiness Level'] == "High"]
data
Above, you can see we've added a column showing the general happiness level for each country (high, medium, or low).
# labels for x axis
factors = ['GDP', 'Social Support', 'Healthy Life Expectancy', 'Freedom'
, 'Generosity', 'Absence of Corruption']
Let's create boxplots with Matplotlib, comparing the importance of different attributes, based on happiness levels.
# boxplot comparing attributes for low happiness countries
val = [low_happiness['GDP Importance'].values.tolist()
, low_happiness['Social Support Importance'].values.tolist()
, low_happiness['Healthy Life Expectancy Importance'].values.tolist()
, low_happiness['Freedom Importance'].values.tolist()
, low_happiness['Generosity Importance'].values.tolist()
, low_happiness['Absence of Corruption Importance'].values.tolist()]
plt.boxplot(val, patch_artist=True)
plt.xticks([1, 2, 3, 4, 5, 6], factors, rotation=90)
plt.xlabel('Factor Contributing to Happiness')
plt.title('Importance of Factors for Countries with Low Happiness Rankings')
plt.show()
# boxplot comparing attributes for medium happiness countries
val = [medium_happiness['GDP Importance'].values.tolist()
, medium_happiness['Social Support Importance'].values.tolist()
, medium_happiness['Healthy Life Expectancy Importance'].values.tolist()
, medium_happiness['Freedom Importance'].values.tolist()
, medium_happiness['Generosity Importance'].values.tolist()
, medium_happiness['Absence of Corruption Importance'].values.tolist()]
plt.boxplot(val, patch_artist=True)
plt.xticks([1, 2, 3, 4, 5, 6], factors, rotation=90)
plt.xlabel('Factor Contributing to Happiness')
plt.title('Importance of Factors for Countries with Medium Happiness Rankings')
plt.show()
# boxplot comparing attributes for high happiness countries
val = [high_happiness['GDP Importance'].values.tolist()
, high_happiness['Social Support Importance'].values.tolist()
, high_happiness['Healthy Life Expectancy Importance'].values.tolist()
, high_happiness['Freedom Importance'].values.tolist()
, high_happiness['Generosity Importance'].values.tolist()
, high_happiness['Absence of Corruption Importance'].values.tolist()]
plt.boxplot(val, patch_artist=True)
plt.xticks([1, 2, 3, 4, 5, 6], factors, rotation=90)
plt.xlabel('Factor Contributing to Happiness')
plt.title('Importance of Factors for Countries with High Happiness Rankings')
plt.show()
From the three graphs, we can see that social support ranks highest in all three categories. Therefore, regardless of a country's current happiness level, social support is most important to them and is the greatest contribution to happiness across the world.
However, looking at other categories, there seems to be more variability. GDP, for example, is one of the most variable factors. For countries with high happiness, the GDP seems to be almost as important as social support. However, as we look at countries with decreasing happiness levels, it is not as clear. With medium happiness, countries still hold GDP quite high behind social support, but it is lower than the plot in high happiness. When we look at low happiness however, the importance of GDP is significantly lower, and is now tied with healthy life expectancy. Thus, we can conclude that countries that are happier tend to place more importance in GDP.
The other attributes (life expectancy, freedom, generosity, and absence of corruption) also factor into countries' happiness, but rank about the same in all three levels.
Next, we want to show another way to display our data, through linear regression! Using Matplotlib, we will graph each factor against happiness. This is a great way to see the general pattern of our data and to compare attributes. A greater slope means a stronger correlation in data, so we keep an eye out for that when working with linear regression. We are displaying two graphs in order to make things less cluttered - easier for analysis!
factors = data.iloc[:,:9]
fig, axs = plt.subplots(2,figsize=(15,15))
axs[0].set_xlim([-.25,1.75])
axs[1].set_xlim([-.25,1.75])
axs[0].set_ylabel('Happiness')
axs[1].set_ylabel('Happiness')
axs[0].set_xlabel('Importance')
axs[1].set_xlabel('Importance')
axs[0].set_title('Effect of Social Support, Freedom to Make Choices, Absence of Corruption Importance on Happiness')
axs[1].set_title('Effect of GDP, Healthy Life Expectancy, Generosity Importance on Happiness')
# split the factors between the two subplots to clearly
# see the different factors better
for i in range(3,9):
if i % 2 == 0:
plot = 0
else:
plot = 1
axs[plot].scatter(x = factors.iloc[:,i], y = factors["Happiness Score"], label=data.columns[i])
m, b = np.polyfit(factors.iloc[:,i], factors["Happiness Score"],1)
axs[plot].plot(factors.iloc[:,i], m*factors.iloc[:,i] + b, label='y={:.2f}x+{:.2f}'.format(m,b))
axs[0].legend()
axs[1].legend()
In our first graph, we are comparing the importance of social support, freedom, and absence of corruption. The lines are quite similar, but the key/legend we included shows that absence of corruption has the strongest correlation. In our second graph, it is much clearer that healthy life expectancy importance has the strongest correlation compared to GDP and generosity. While the difference was clear in the second plot, the differences in the first plot were harder to see, so including a key with additional information can be incredibly important for accurate data analysis.
Looking at the graphs together, the importance of absence of corruption leads with the strongest correlation. This means that as countries get happier, they tend to place more significance on their lack of corruption, more so than other factors like GDP, life expectancy, etc. The importance of generosity had the weakest correlation, indicating that generosity is the least closely tied to a country's happiness. However, it certainly does not mean that absence of corruption is the most important, or that generosity is the least important to a country's happiness score.
We also want to test if there is a relationship between actual factors, like true GDP and true life expectancy, on their importance. So, does a country's actual GDP affect how much weight a country gives it, and how does that affect happiness? We want to test for the same for life expectancy, as well.
First, let's organize this data:
exp = data[['Country','Happiness Score','GDP Importance','Actual GDP','Healthy Life Expectancy Importance','Actual Life Expectancy']]
exp = exp.rename(columns={'Healthy Life Expectancy Importance':'Life Expectancy Importance',
'Happiness Score':'Happiness'})
exp["Life Expectancy Levels"] = pd.cut(exp["Actual Life Expectancy"], 3, labels=["Low", "Medium", "High"])
exp
Now, let's graph it!
fig, axs = plt.subplots(2,2,figsize=(15,15))
ind = 0
for factor in ['GDP', 'Life Expectancy']:
# create our two graphs comparing real life circumstance to its weighted importance
axs[ind][0].scatter(x = exp['Actual ' + factor], y = exp[factor + ' Importance'])
axs[ind][0].set_title('Effect of Actual ' + factor + ' on ' + factor + ' Importance')
axs[ind][0].set_xlabel('Actual ' + factor)
axs[ind][0].set_ylabel(factor + ' Importance')
for level in ['High','Medium','Low']:
# for both GDP and Life Expectancy, graph all three levels of real life
# circumstance in different colors and label them
axs[ind][1].scatter(x = exp[exp['Life Expectancy Levels'] == level][factor + ' Importance'],
y = exp[exp['Life Expectancy Levels'] == level]['Happiness'], label = level + ' ' + factor)
axs[ind][1].set_title('Effect of ' + factor + ' Importance on Happiness')
axs[ind][1].set_ylabel('Happiness')
axs[ind][1].set_xlabel(factor + ' Importance')
ind = ind + 1
axs[0][1].legend()
axs[1][1].legend()
The plots on the left show how actual GDP and life expectancy affect their importance. The curve of the first graph indicates that some countries with lower GDP do not find it important; however, as GDP increases, they place more value in it. For life expectancy, the clear linear path shows that as life expectancy increases, it becomes more important.
The righthand plots show that as actual GDP and life expectancy increase, countries become happier. So, generally, as GDP and life expectancy increase, countries think of them as more important and it contributes to their overall happiness. Especially with life expectancy, there seems to be a strict divide in importance depending on whether or not countries have high life expectancies. GDP has a similar trend, but with blurred boundaries. Although, because there are no data points with cominations such as high life expectancy importance and low life expectancy, it's hard to see whether high importance with low reality leads to unhappiness, or any similar hypothesis.
Prediction and Machine Learning¶
For this section, we are using statsmodels:
import statsmodels.formula.api as sm
First, we want to prepare a new dataframe with the our old dataframe's names with no spaces, because of the format that statsmodels takes in attributes to create their linear regressions. An 'All' columns is created for Mean Squared Error analysis later on and will be used as the attribute to group all elements by.
model = (data.rename(columns={'Happiness Score':'Score','GDP Importance':'GDPPerCapita', 'Social Support Importance':'SocialSupport','Healthy Life Expectancy Importance':'HealthLifeExpectancy',
'Freedom Importance':'FreedomToMakeLifeDecisions', 'Absence of Corruption Importance':'PerceptionsOfCorruption',
'Generosity Importance':'Generosity','Actual GDP':'RealGDP','Actual Life Expectancy':'RealLifeExpectancy'})
.iloc[:,2:])
# Create an all attribute with the value 1 for every entity
model['All'] = [1] * len(model['Score'])
For future reference, a clean representation of the current column names we now have:
model.columns
Now, to get the multi-dimensional linear regression:
# Use statsmodel to calculate a linear regression
multi_res = sm.ols('Score~1+GDPPerCapita+SocialSupport+HealthLifeExpectancy+FreedomToMakeLifeDecisions+Generosity+PerceptionsOfCorruption', data=model).fit()
multi_res.summary()
When analyzing these summary tables, the coefficients of each attribute is a good start to see the correlation the model draws between, in our case, Happiness Score and each of these attributes. At a glance, you can see all of them have positive coefficients, meaning that as the importance of all of these factors increase, the happier a person is expected to be. The next thing we can look at is P-score. When P-score is < 0.05, a relationship is able to be called significant. In this case, it looks like all but Generosity and Perception of Corruption have a significant relationship to Happiness Score. If you recall, our previous scatter plots with linear regressions of each attribute seperately also seemed to show that the despite consistently low generosity and corruption importance, Happiness Scores had a large range, which makes sense under these findings as well.
To better understand how well this model creates Happiness Score predictions, let's create a prediction for each country to compare with the actual score. While visualizations are always most helpful, the multi-dimensioinality of our data makes such a task hard, so instead we'll analyze our data through mean squared error. To calculate it, we'll be computing the difference squared as an attribute and aggregating to get the mean, as shown below.
# First, create a dataframe consisting of the attributes included in the linear regression
multi = model[['Score','GDPPerCapita','SocialSupport','HealthLifeExpectancy','FreedomToMakeLifeDecisions','Generosity','PerceptionsOfCorruption']]
# Use the regression to predict a happiness score
prediction = multi_res.predict(multi)
# Add predictions and difference squared to our model dataframe
model['Multi Res'] = prediction
model['MR Difference Squared'] = (model['Score'] - model['Multi Res']) ** 2
model[['Score','Multi Res','MR Difference Squared']].head()
# Calculate Mean Squared Error
mse = model.groupby('All').agg({'All':'count','MR Difference Squared':'mean'}).rename(columns={'MR Difference Squared':'MR MSE'})
mse
Now that we've gained more insight into how well the linear regression predicts, let's see if there's some kind of interaction between real GDP and life expectancy and importance when it comes to predicting happiness.
interact_res = sm.ols('Score~GDPPerCapita*RealGDP+HealthLifeExpectancy*RealLifeExpectancy',data=model).fit()
interact_res.summary()
Unfortunately, it seems like only GDP importance and real life expectancy have a significant relationship under this model, with incredibly high P-scores for both interactions as well as other stand alone attributes. Sadly,this means that we cannot make any claims about the interaction between real life circumstance and perception of importance when it comes to happiness, but learning there is no relationship is still learning something and remains an important conclusion that can be drawn though machine learning! To further understand the accuracy of this model's results in a comparable metric to our first, we'll repeat the same Mean Squared Error analysis.
multi = model[['Score','GDPPerCapita','RealGDP','HealthLifeExpectancy','RealLifeExpectancy']]
prediction = interact_res.predict(multi)
model['Interact Res'] = prediction
model['IR Difference Squared'] = (model['Score'] - model['Interact Res']) ** 2
model[['Score','Multi Res','MR Difference Squared','Interact Res','IR Difference Squared']].head()
mse = model.groupby('All').agg({'All':'count','IR Difference Squared':'mean'}).rename(columns={'IR Difference Squared':'IR MSE'})
mse
It looks as if, on average, this interaction model does worse at predicting than our first model. Although, before picking the best of these two, we'd like to try one last model.
But! before we move on, let's take a quick break from our regular happiness-model programming to do some one-dimensional models testing the relationships we saw between actual GDP and life expectancy and their importance to people. And as we do some one dimensional modeling, let's visualize our regressions!
GDP_res = sm.ols('GDPPerCapita~RealGDP', data=model).fit()
GDP_res.summary()
le_res = sm.ols('HealthLifeExpectancy~RealLifeExpectancy', data=model).fit()
le_res.summary()
Looks like we have small P-scores and significant relationships for both life expectancy and GDP pairs, now to visualize our regressions using predictions:
le = pd.DataFrame({'RealLifeExpectancy': [model.RealLifeExpectancy.min(), model.RealLifeExpectancy.max()]})
preds = le_res.predict(le)
preds
# first, plot the observed data
model.plot(kind='scatter', x='RealLifeExpectancy', y='HealthLifeExpectancy')
# then, plot the least squares line
plt.plot(le, preds, c='red', linewidth=2)
plt.title('Effect of Real Life Expectancy on its Importance')
gdp = pd.DataFrame({'RealGDP': [model.RealGDP.min(), model.RealGDP.max()]})
preds = GDP_res.predict(gdp)
preds
# first, plot the observed data
model.plot(kind='scatter', x='RealGDP', y='GDPPerCapita')
# then, plot the least squares line
plt.plot(gdp, preds, c='red', linewidth=2)
plt.title('Effect of Real GDP on its Importance')
The linear regression looks to match life expectancy pretty well, while GDP is a little bit more of a stretch, so before we return to our main happiness plots, take this as a lesson to use visualization and further analysis when possible, because other metrics such as the P-scores of our model may not be giving the full picture. Beyond visualization, it also looks like Mean Squared Error analysis would've revealed a less than perfect regression, but we'll leave that to you to try and calculate! Does the GDP regression perform significantly worse than the life expectancy regression?
Now, back to happiness models. The final thing we'd like to try is one last multi-dimensional linear regression using only the attributes that got a P-score less than 0.05 from our first model.
selective_res = sm.ols('Score~1+GDPPerCapita+SocialSupport+HealthLifeExpectancy+FreedomToMakeLifeDecisions',data=model).fit()
selective_res.summary()
All the P-values are less than 0.05! But, let's do the same Mean Squared Error analysis to see how predictions line up.
select = model[['Score','GDPPerCapita','SocialSupport','HealthLifeExpectancy','FreedomToMakeLifeDecisions']]
prediction = selective_res.predict(select)
model['Selective Res'] = prediction
model['SR Difference Squared'] = (model['Score'] - model['Selective Res']) ** 2
model[['Score','Multi Res','MR Difference Squared','Interact Res','IR Difference Squared','Selective Res','SR Difference Squared']].head()
mse = model.groupby('All').agg({'All':'count','SR Difference Squared':'mean'}).rename(columns={'SR Difference Squared':'SR MSE'})
mse
It looks neck and neck, so the final thing we'll do is see how the distributions of error for each model look with a Box plot visualization!
val = [model['MR Difference Squared'].values.tolist()
, model['IR Difference Squared'].values.tolist()
, model['SR Difference Squared'].values.tolist()]
plt.boxplot(val, patch_artist=True)
plt.xticks([1, 2, 3], ['Multi-Res','Interact-Res','Selective-Res'], rotation=45)
plt.xlabel('Linear Model')
plt.ylabel('Difference Squared')
plt.title('Difference Squared Distributions across Different Linear Models')
plt.show()
Although they look fairly similar, Multi-Res not only shows a lower distribution, but it also looks more concentrated towards lower differences squared, and has lower outliers. From this, we can use our multi-res model to draw our final conclusions and wrap up this tutorial!
Summary/Conclusion¶
From all this, we've seen that how we weigh the importance of factors such as GDP, life expectancy, social support, and freedom, have a big impact on how happy we are. On top of this, although we can't say whether the interaction of circumstance and perception affect happiness and how to optimize such relationships, we do know that better circumstances lead to people giving such factors more importance, which is seen to increase happiness scores. Though, beyond these happiness insights, we hope we've left you with all the tools you need to make some of your own! From data cleaning to exploratory analysis to prediction and machine learning, we hope you've been able learn how to create your own pipeline and use any of these libraries that have been newly introduced to you. And finally, just as data science has endless possibilities, we hope you don't stop here as you open yourself up to different ways of visualizing data, such as going beyond 2D representations, and different machine learning techniques such as decision trees and random forests. As you continue your explorations, we hope this was a helpful start!
If you want to learn even more about data science, feel free to check out these resources: